Reinforcement Learning in Continuous Time: Advantage Updating

A new algorithm for reinforcement learning, advantage updating, is described. Advantage updating is a direct learning technique; it does not require a model to be given or learned. It is incremental, requiring only a constant amount of calculation per time step, independent of the number of possible actions, possible outcomes from a given action, or number of states. Analysis and simulation indicate that advantage updating is applicable to reinforcement learning systems working in continuous time (or discrete time with small time steps) for which standard algorithms such as Q-learning are not applicable. Simulation results are presented indicating that for a simple linear quadratic regulator (LQR) problem, advantage updating learns more quickly than Q-learning by a factor of 100,000 when the time step is small. Even for large time steps, advantage updating is never slower than Q-learning, and advantage updating is more resistant to noise than is Q-learning. Convergence properties are discussed. It is proved that the learning rule for advantage updating converges to the optimal policy with probability one. REINFORCEMENT LEARNING A reinforcement learning problem is an optimal control problem where the controller is given a scalar reinforcement signal (or cost function) indicating how well it is performing. The reinforcement signal is a function of the state of the system being controlled and the control signals chosen by the controller. The goal is to maximize the expected total discounted reinforcement, which for continuous time is defined as E γ t r ( x t , u t ) dt